This document discusses using DASH (Dynamic Adaptive Streaming over HTTP) to deliver rich media services with synchronized audio, video, text, and graphics. It proposes two studies: A) providing an API for manipulating DASH-delivered audio/video in rich media documents while maintaining compatibility with existing languages, and B) delivering rich media streams synchronously with related audio/video by repackaging existing broadcast services over DASH to minimize bandwidth and preserve interactivity. The implementation is discussed using open source GPAC tools to support DASH playback, file segmentation, and muxing.

1. Usages of DASH for Rich Media Services Cyril Concolato Jean Le Feuvre RomainBouqueau MMSys 2011

2. Rich Media Services Services featuring Multiple media elements audio, video, text, 2D/3D graphics Dynamicity Changing text, animated graphics, … Possibly triggered on server-side (streaming) Synchronization with media elements Interactivity Example of Rich Media Technologies HTML 5, MPEG-4 BIFS, Flash… page 1

3. Context Rapid development of HTTP Streaming technologies MPEG, 3GPP, OIPF, W3C, … Increased usages of Rich Media Technologies Presentation glue for A/V content (HTML+video) Streamed services including live meta-data (SVG/DIMS, MPEG-4 BIFS) page 2

4. Research Questions How can Rich Media languages use DASH-delivered audio/video content? Study A How can Rich Media content be delivered over DASH? Study B page 3

5. Study A: Description Goal Provide a generic API for MPD manipulations within a rich media document Compatibility with existing Rich Media languages Hypothesis Bandwidth related manipulations are handled at the DASH level Trick Mode related manipulations may be handledat both DASH and browser levels Requirements The API shall provide access to: The desired video resolution & frame rate The desired language The desired quality The desired view page 4

6. Study A: Architecture page 5 Browser Media Player IntegratedRendering Media Elements (audio,video…) Media Selectionand Control Rich Media Elements (HTML, …) DASH Client HTTP Stack

7. Study A: Solutions Use of fragment identifiers Use of attributes in DASH namespace Use of ECMAScript page 6 <video src="dash.mpd#viewpoint=1&width=176&height=144"> <video src="dash.mpd"> <track kind=subtitles src="dash.mpd" dash:qualityRanking="1" srclang="en"> </video> var track = createTrackFromDASH(‘subtitles’,‘lang’,’en’);

8. Study B: Description Goal DeliverRich Media Streamssynchronouslywithrelated audio/video data Automaticrepackaging of existingbroadcast interactive services over DASH Requirements Enable transposition of the traditional carrousel approach Minimizebandwidth, leverage HTTP Preserveinteractivity Use case Digital Radio Service [demonstration: http://www.youtube.com/watch?v=Bmer91TZhCo] T-DMB (MPEG-2 TS + MPEG-4 A/V + MPEG-4 BIFS) page 7

9. Study B: Architecture page 8 IntegratedRendering Browser Media Player Rich Media Elements (HTML, …) Media Elements(audio, video…) DASH Client HTTP Stack

10. page 9 Digital Radio Service Broadcast environment Digital radio stream Audio Data carousel (500 ms) Data carousel (2 s) Live data Digital radio receivers

12. Approach 3: External data references page 12 Digital radio stream segments … Media Segment with updated data references Media Segment Media Segment with initial datareferences with initial datareferences = HTTP link to = HTTP link to = HTTP link to

14. AdditionalConsideration:RedundantSignaling page 14 Media Segment #n Media Segment #n+1 No change in « carousel » data User interactswith Client 1 and locally changes the Rich Media State Client 1 requests segment #n+1 Client 1 starts and requests segment #n Client 1 processes segment #n including « Carousel » data Client 1 ignores redundant data (preserved interaction) Client 2 starts andrequests segment #n+1 Client 2 processes « Carousel » data

15. Study B: Summary of solutions Use of data in Initialization Segment (IS) Under evaluation for DASH Use of external data references (EDR) Requires MP4 Under evaluation for DASH Use of redundantsignaling (RS) in MP4 for RM data Accepted as an MP4 amendment Possible joint usages: IS+EDR or EDR + RS or IS+EDR+RS page 15

16. Implementation GPAC Open Source Projet DASH Player Supports MPD and M3U8 playlists MP4Box Segmentation tool for MP4 files MP42TS DVB & DMB Mux Segmentation page 16

17. Thank youQuestions ? Cyril.Concolato@telecom-paristech.fr Jean.LeFeuvre@telecom-paristech.fr Romain.Bouqueau@telecom.paristech.fr http://gpac.sourceforge.net